
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark OfHce 
Address: COMMISSIONER FOR PATENTS 
P.O.Box 1450 

Alexandria. Vir^gima 22313-1450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/739,952 


12/18/2000 


Junichi Rekimoto 


112857-251 


4829 



29175 7590 01/16/2004 

BELL, BOYD & LLOYD, LLC 
P. O. BOX 1135 
CHICAGO, IL 60690-1135 



EXAMINER 



CHUONO, TRUCT 



ART UNIT 



PAPER NUMBER 



2174 

DATE MAILED: 01/16/2004 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO-90C (Rev. 10/03) 



Office Action Summary 



Application No. 

09/739,952 



Examiner 

True T Chuong 



Appllcant(s) 

REKIMOTO. JUNICHI 



Art Unit 

2174 



The MAILING DATE of this communication appears on the cover sheet with the correspond nee address - 
Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

- If the period for reply specified above is less than thirty (30) days, a reply wWh'in the statutory minimum of thirty (30) days will be considered timely. 

- If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
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DETAILED ACTION 



Election/Restrictions 

1. Applicant's election without traverse of Group II (Claims 1-13, 15-28, and 30-35) in 
Paper No. 8 is acknowledged. 

Claim Objections 

Claims 1-13, 15-28, and 30-35 are objected to because of the following informalities: in 
claim 1, "stationarily" should be "stationary". Appropriate correction is required. 

Other claims also have the similar problem, and all dependent claims are also objected 
because of their dependency. 

Claim Rejections - 35 (JSC §102 
1, The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form.the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

' (e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 35 1(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 2 1(2) of such treaty in the English language. 

3. Claims 1-13, 15-28, and 30-35 are rejected under 35 U.S.C. 102(e) as being anticipated 



by Filo et al. (U.S. Patent No. 6,215,498 Bl). 
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As to claim 1, Filo teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space (col. 2 line 
55-col. 3 line 37, figs. 5A-B, 7-10); 

a display means for displaying an image on said operation surfaces (col. 3 lines 1-49); 

an imaging means for picking up an image of said operation surfaces (canned images, 
col 4 lines 1-34); 

an environment type computer stationarily arranged in said information space (col. 3 
lines 1-37 and col. 4 lines 1-57); and 

one or more than one physical objects mounted on said operation surfaces and containing 
a visually identifiable visual marker on the surface thereof (col. 7 lines 1-30, identification 
levels, col. 9 line 1-22 and col. 10 lines 10-61); 

said environmental type computer being adapted to execute (col 13 lines 1-42); 

a processing operation of recognizing the identification information and the position 
information of each of said physical objects in said visual marker (col 13 line 43-col 14 lines 
33); 

a processing operation of recognizing the digital object dropped to a site on the surface of 
each of said physical objects (col 15 line 29-col 16 line 60, and figs. 10-1 1 and 13); and 

a processing operation of forming link information for linking the digital object to the 
dropped site on the surface for each of said physical objects (col. 9 lines 1-23, col 15 line 29-col 
16 line 60, and figs. 10-11 and 13). 
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As to claim 2, Filo teaches the information input/output system according to claim 1, 
wherein at least one of said physical objects is a portable computer capable of being moved in 
said information space and exchanging digital objects with other computers (canned images, col. 
9 lines 1-23, col. 13 line 43-col. 14 line 24). 

As to claim 3, Filo the information input/output system according to claim 1, wherein 

at least one of said physical objects is a portable computer capable of being moved in said 
information space and exchanging digital objects with other computers (canned images, col. 9 
lines 1-23, col. 13 line 43-col. 14 line 24); and 

said environment type computer can expand a mouse/cursor operation on the installed 
portable computer onto said operation surfaces (point and click, col 3 lines 1-49). 

As to claim 4, Filo teaches the information input/output system according to claim 1, 
wherein 

said imaging means can identify the position indicated by an optical pointer for indicating 
a specific position by irradiating a beam of light with a predetermined wavelength (three 
dimensional view of the distance, col. 1 1 lines 27-66); and 

the user is allowed to indicate a position in said information space by means of 
coordinates and the optical pointer (col. 1 1 line 12-col. 12 line 21). 

As to claim 5, Filo teaches the information input/output system according to claim 1, 
wherein said environment type computer executes a processing operation of calling the linked 
digital object and/or displaying the digital object to the user in response to a user operation 
applied to the site of forming the link information on the surface of each of the physical objects 
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on the basis of the image picked up by said imaging means (col. 15 line 29-col. 16 line 60, and 
figs. 10-11 and 13). 

As to claim 6, Filo teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; an environment type computer stationarily arranged in said 
information space; one or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof (note the rejection of claim 
1 above); and 

a portable ID recognition device adapted to recognize/identify the visual marker on the 
surface of each of said physical objects (col. 3 lines 1-37 and col. 4 line 1-22). 

As to claim 7, Filo teaches the information input/output system according to claim 6, 
wherein said environmental type computer is adapted to execute; a processing operation of 
receiving the identification information of the source object and that of the destination object 
from said ID recognition device (col. 7 lines 1-30, identification levels, col. 9 line 1-22 and col. 
10 lines 10-61); and a processing operation of applying an action specified on the basis of the 
combination of the type of the source object and that of the destination object (col. 3 line 26 -col. 
4 line 21). 

As to claim 8, Filo teaches the information input/output system according to claim 6, 
wherein at least one of said physical objects is a portable computer capable of being moved in 
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said information space and exchanging digital objects with other computers (col. 9 lines 1-23, 
col. 13 line 43-col. 14 line 24). 

As to claim 9, Filo teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; an environment type computer stationarily arranged in said 
information space; and one or more than one physical objects mounted on said operation surfaces 
and containing a visually identifiable visual marker on the surface thereof (note the rejection of 
claim 1 above); 

at least one of said physical objects being a virtual camera having a virtual imaging 
direction as specified by the position information contained in the visual marker (cameras, col. 9 
lines 1-23, col. 13 line 43-col. 14 line 24). 

As to claim 10, Filo teaches the information input/output system according to claim 9, 
wherein 

said environmental type computer is adapted to execute (col. 9 lines 1-39); 
a processing operation of recognizing/identifying said virtual camera and identifying the 
position information and the virtual imaging direction (col. 10 lines 10-61 and col. 14 lines 10- 



a processing operation of generating a virtual picked up image according to the position 
information and the virtual imaging direction (figs. 10-11); and 



32); 




Application/Control Number: 09/739,952 Page 7 

Art Unit: 2174 

a processing operation of displaying the virtual picked up image to the user (col. 14 lines 
10-32, figs. 7, 10 and 11). 

As to claim 11, Filo teaches the information input/output system according to claim 9, 
wherein at least one of said physical objects is a portable computer that can move in said 
information space and exchange digital objects with another computer (col. 9 lines 1-23, col 13 
line 43-col. 14 line 24, and figs. 7, 9-1 1). 

As to claim 12, Filo teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; an environment type computer stationarily arranged in said 
information space; one or more than one physical objects mounted on said operation surfaces and 
containing a visually identifiable visual marker on the surface thereof (note the rejection of claim 
1 above); and 

an optical pointer adapted to point a specific position by irradiating a beam of light with a 
predetermined wavelength (three dimensional view of the distance, col 1 1 lines 27-66); 

said imaging means being capable of identifying the position pointed by said optical 
pointer (col 11 line 12-col 12 line 21). 

As to claim 13, Filo teaches the information input/output system according to claim 12, 
wherein 
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said environment image type computer executes on the basis of the picked up by said 
imaging means (canned images, col. 4 lines 1-34); 

a processing operation of recognizing the user operation on said operation surfaces, using 
said optical pointer (col. 1 1 line 12-col. 12 line 21); and 

a processing operation of controlling the display of the digital object by said display 
means according to the result of the recognition (col 13 lines 43-67). 

As to claim 15, Filo teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information pace; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; an environment type computer stationarily arranged in said 
information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; one or more than one physical objects mounted on said operation 
surfaces and containing a visually identifiable visual marker on the surface thereof (note the 
rejection of claim 12 above); and 

a portable ID recognition device adapted tO' recognize the visual marker on the surface of 
each of said physical objects and being capable of holding or releasing the recognized 
identification information (col. 3 lines 1-37 and col. 4 line 1-22). 

As to claim 16, Filo teaches the information input/output system according to claim 15, 
wherein 
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said environment type computer executes; a processing operation of obtaining a digital 
object corresponding to the held identification information in response to the holding operation 
of said ID recognition device (col. 3 lines 1-37 and col. 4 line 1-22); and 

a processing operation of transferring a digital object corresponding to the held 
identification information to a nearby physical object in response to the releasing operation of 
said ID recognition device (col. 9 lines 1-23, and col. 10 lines 1-61). 

As to claim 17, Filo teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said system comprising: 

one or more than one operation surfaces arranged in said information space; a display 
means for displaying an image on said operation surfaces; an imaging means for picking up an 
image of said operation surfaces; an environment type computer stationarily arranged in said 
information space; one or more than one portable computers capable of exchanging digital 
objects with other computers; and one or more than one physical objects mounted on said 
operation surfaces and containing a visually identifiable visual marker on the surface thereof 
(note the rejection of claim 12 above); 

said environment type computer being adapted to allow mouse/cursor operations on said 
portable computers to extend to said operation surfaces and display a rubber band as visual 
feedback according to the area on said operation surfaces as defined by means of said 
mouse/cursor (col, 10 lines 1-62, and figs. 7, 9-11). 

As to claim 18, Filo teaches the information input/output system according to claim 17, 
wherein said environment type computer is adapted to pick up an image of the area defined by 
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the rubber band by means of said imaging means and take the picked up image in said 
information space as digital object (col. 10 lines 1-62, and figs. 7, 9-1 1). 

As to claim 19, Fito teaches the information input/output system according to claim 17, 
wherein another object can apply a process or a method it possesses to the rubberbanded original 
digital object in response to an action of said original object of being dropped on said another 
object (col. 3 lines 1-37, col. 4 lines 1-21, col. 10 lines 1-62, col. and figs. 7, 9-11). 

As to claim 20, Fito teaches the information input/output system according to claim 19, 
wherein said another object is a physical object possessing a process or a method for printing a 
document (printing documents, col. 10 lines 52-61). 

As to claim 21, Fito teaches the information input/output system according to claim 19, 
wherein said another object is a name card possessing a process or a method for mailing a 
document to the mail address corresponding to the name on it (e-mail documents, col. 15 lines 
55-67). 

As to claim 22, Fito teaches an information input/output system to be used for user 
operations relating to an object in an information space realized by expanding a digital space of a 
computer into the real world, said information space having one or more than one operation 
surfaces arranged in said information space, a display means for displaying an image on said 
operation surfaces, an imaging means for picking up an image of said operation surfaces, an 
environment type computer stationarily arranged in said information space and one or more than 
one physical objects mounted on said operation surfaces and containing a visually identifiable 
visual marker on the surface thereof, said system comprising: 
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a step of firstly recognizing the identification information and the position information of 
each of said physical objects from the visual marker (canned images, col. 9 lines 1-23, col. 13 
line43-col, 14 line 24); 

a step of secondly recognizing the digital object dropped to a site on the surface of each 
of said physical objects (col. 9 lines 1-23, col. 13 line 43-col. 14 line 24, and figs. 7, 9-1 1); and 

a step of forming link information for linking the digital object to the dropped site on the 
surface for each of said physical objects (link objects, col. 9 lines 1-23, col. 15 line 29-col. 16 
line 60, and figs. 10-11 and 13). 

As to claim 23, Filo teaches the information input/output system according to claim 22, 
wherein 

at least one of said physical objects is a portable computer capable of being moved in said 
information space and exchanging digital objects with other computers (note the rejection of 
claim 12 above); and 

said environment type computer can expand a mouse/cursor operation on the installed 
portable computer onto said operation surfaces in said second recognition step (col. 9 lines 1-23, 
col. 13 line 43-col. 14 line 24, and figs. 7, 9-11). 

As to claim 24, it is individually similar in scope to claim 4 above; therefore, rejected 
under similar rationale. 

As to claim 25, Filo teaches the information input/output system according to claim 22, 
further comprising: 

a step of calling the linked digital object and/or displaying the digital object to the user in 
response to a user operation applied to the site of forming the link information on the surface of 
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each of the physical objects (link objects, col. 9 lines 1-23, col. 15 line 29-col. 16 line 60, and 
figs. 10-11 and 13). 

As to claim 26, it is individually similar in scope to claim 7 above; therefore, rejected 
under similar rationale. 

As to claim 27, it is individually similar in scope to claim 1 above; therefore, rejected 
under similar rationale. 

As to claim 28, it is individually similar in scope to claim 13 above; therefore, rejected 
under similar rationale. 

As to claim 30, it is individually similar in scope to claim 16 above; therefore, rejected 
under similar rationale. 

As to claim 3 1, it is individually similar in scope to claim 17 above; therefore, rejected 
under similar rationale. 

As to claim 32, it is individually similar in scope to claim 18 above; therefore, rejected 
under similar rationale. 

As to claim 33, it is individually similar in scope to claim 19 above; therefore, rejected 
under similar rationale. 

As to claim 34, it is individually similar in scope to claim 20 above; therefore, rejected 
under similar rationale. 

As to claim 35, it is individually similar in scope to claim 21 above; therefore, rejected 
under similar rationale. 



Application/Control Number: 09/739,952 
Art Unit: 2174 



Page 13 



Conclusion 



4. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

Dementhon et al. (U.S. Patent No. 5,297,061) teach visual displays, camera, drag 
and drop objects from/to different screens, links, and identification (cols. 2-12 and figs. 1, 
3, Sand 11). 

Platzker et al (U.S. Patent No. 5,528,263) teach GUI, 3D, camera, moving 
objects, links, and pointing devices (cols. 2-10 and figs. 1-5B). 

Card et al. (U.S. Patent No. 5,838,326) teach 3D, drag and drop objects, id, and 
GUI (cols. 2-10 and figs. 2A-12). 

Parsons (U.S. Patent No. 6,025,844) teaches drag and drop objects, GUI, target, 
different computers, and link relationship (cols. 4-12 and figs. 10-19D). 

5, Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to True T Chuong v^hose telephone number is 703-305-5753. The 
examiner can normally be reached on M-Th and alternate Fridays 8:30 AM - 5:00 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Kristine L. Kincaid can be reached on 703-308-0640. The fax phone number for the 
organization where this application or proceeding is assigned is 703-746-7239. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is 703-305-3900. 



KRISTINE KINCAID 
SUPERVISORY PATENT EXAMINER 
TECHNOLOGY CENTER 2100 



True T. Chuong 
01/09/04 



